Repository navigation
feat(database)!: replace postgres with sqlite - #61
Merged
Merged
Conversation
The bot stores its data for a single guild, and each instance was given its own postgres database to hold it. That cost a database server per instance: a second container to pull and run, credentials to configure, a healthcheck to wait on, and around 45 MiB of resident memory that the bot did not need. The data itself is a handful of rows per guild, so a file is enough. The database is now SQLite, through modernc.org/sqlite. The driver is pure Go, which keeps the build at CGO_ENABLED=0 and the runtime image unchanged. It binds $1-style placeholders by index the way postgres does, so the query text carried over without being rewritten, and it binds them correctly: mattn/go-sqlite3 and ncruces/go-sqlite3 both bind by order of appearance, which would have put the wrong values in the wrong place in the queries that build their argument list. Timestamps are stored as Unix milliseconds in INTEGER columns. SQLite has no timestamp type, and the obvious alternative is silently wrong: the driver writes a time.Time as text while a column default writes an integer, so the column holds two kinds of value and SQLite orders it by kind rather than by instant. Storing integers and letting the driver convert keeps the column and the comparisons exact. The subtleties of that representation are written down in AGENTS.md, since the two ways of getting it wrong are a query that returns rows out of order and one that fails to scan. The schema uses INTEGER PRIMARY KEY AUTOINCREMENT. The AUTOINCREMENT matters: without it SQLite reuses an id after a delete, and the pagination cursor compares (created_at, id), so a reused id would make a page skip rows. The moderation log index ends with id for the same reason, and without it SQLite sorts every page again with a temporary B-tree over the table. A test asserts the query plan, so the sort cannot come back unnoticed. The migration runner hands a file to the driver as one statement instead of splitting it on semicolons. The driver knows the quoting rules, so a semicolon inside a string literal no longer cuts a statement in half. Two indexes that the unique constraints and the composite index already covered are gone. DB_PATH names the file, and DB_BOT_INSTANCE_ID names it when DB_PATH is unset, so instances sharing a directory still get one file each. The instance id is filtered down to characters that are safe in a file name. The image creates the data directory owned by the user it runs as, so a named volume mounted over it is writable without anyone having to fix its permissions by hand. Breaking: DATABASE_URL, DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, DB_NAME and POSTGRES_PASSWORD are gone, replaced by DB_PATH and DB_BOT_INSTANCE_ID. compose.yaml no longer starts a database service. There is no migration path for an existing postgres database. Nothing was deployed against it, so there is nothing to carry over. docs/OPERATIONS/DATABASE.md covers where the file lives, how to back it up, and how to read it. The backup uses VACUUM INTO, because the write-ahead log beside the file holds writes that a plain copy would miss.
analeis
force-pushed
the
migrate-sqlite
branch
from
November 10, 2026 18:39
6592fe2 to
48afbf8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the PostgreSQL backend with SQLite, measured against the database server it replaces.
Both sides were benchmarked with the
internal/dbpackage taken from the respective commit, driven by the same harness over the same public API, on the same 8-core host. PostgreSQL 18.3 ran locally over a unix socket, so it was not charged for network latency. Figures are the median of three runs against a freshly created database, after a warm-up run excluded the cold-start checkpoint storm.Query latency
Startup and bulk load
Concurrent load
SQLite serialises writes, so this is one writer against seven readers. The write-ahead log and a busy timeout keep the readers off the writer's path, and the pool is small on purpose.
Footprint
postgres:16-alpine(297 MiB)DB_USER,DB_PASSWORD,POSTGRES_PASSWORDThe binary grows because the SQLite engine is a pure Go port rather than a C library, which is what keeps the build at
CGO_ENABLED=0and the runtime image unchanged.AI Usage disclosure
OpenCode with MiniMax M3.1 was used to migrate the backend, write the tests and the documentation, and produce the benchmarks above. The driver choice was settled by measuring the three candidates against the placeholder binding this codebase relies on:
mattn/go-sqlite3andncruces/go-sqlite3both bind$1-style placeholders by order of appearance rather than by index, which would have put the wrong values in the wrong place in the queries that build their argument list.The implementation was reviewed or redirected atleast three times after the initial prompt for a thorough evaluation and feasibility of the migration.
sqlc was also evaluated and rejected due to being otherwise feasible but still in beta for SQLite, especially $ values.
Tested with
go test ./...,custom-gclincluding NilAway, and the Sphinx-Wbuild, plus the container image built and run against a named volume.